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Executive  Summary 


This  research  tasks,  focused  on  an  assessment  of  systems  engineering  research  needs  and 
workforce  development  assessment,  involved  a  data  collection  exercise  resulting  from  24  site 
visits  (19)  and  telephone  discussions  (5).  Additionally,  we  collaborated  with  INCOSE  to  run  a 
supporting  survey  on  this  topic  to  all  the  INCOSE  Fellows  -  worldwide  (approximately  30%  of  all 
Fellows  responded  to  the  survey).  Flere  is  the  list  of  these  24  engagements: 

1.  Sandia  National  Laboratories  -  Engagement  1 

2.  Sandia  National  Laboratories  -  Engagement  2 

3.  Jet  Propulsion  Laboratory 

4.  NSWC  -  Corona  Division 

5.  NSWC  -  Philadelphia  Division 

6.  Wright  Patterson  AFB-AFLCMC 

7.  Naval  Air  Weapons  Station  -  China  Lake 

8.  AMSAA- Aberdeen  Proving  Grounds 

9.  Embedded  Systems  Institute  -  Netherlands 

10.  The  Aerospace  Corporation 

11.  The  MITRE  Corporation 

12.  AFOTEC- Kirkland  AFB 

13.  NSWC  -  Carderock 

14.  NSWC-NUWC 

15.  NSWC  -  Dahlgren 

16.  NAVSEA  Headquarters 

17.  1C  Engagement  1 

18.  1C  Engagement  2 

19.  INCOSE  Fellows  -  Worldwide  Survey 

20.  ARDEC  -  Picatinny  Arsenal 

21.  NAVSEA  -  SPAWAR 

22.  NAVSEA  05T 

23.  PEO  -  Aviation 

24.  AMRDEC  -  NASA  (Marshall)  -  Joint  Visit 

The  data  collected  suggests  research  priorities  that  broadly  align  with  the  following  4  primary 
research  objectives.  Sub-categories  within  each  primary  research  objective  are  also 
highlighted: 

1.  Knowledge,  Data  and  Machine  Learning 

a)  Al/Machine  Learning; 

b)  Data  Analytics 

c)  Data/Complexity 

d)  Knowledge  Management  and  Document/Requirements 
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2.  Systems  Engineering  MBSE,  Decision  Making  and  Integration. 

a)  Computational  Applications  in  SE 

b)  Decision  Making 

c)  Model  Based  SE 

d)  System  Integration 

e)  Systems  Engineering  Modeling  and  Measurement 

3.  Management,  Culture  and  Agility 

a)  Mission  Engineering 

b)  Agile  Engineering 

4.  Security,  Trust,  Risk  and  Testing 

a)  Cyber-security 

b)  Risk  and  Uncertainty 

c)  Security,  Reliability  and  Resilience 

d)  System  Trust 

e)  System  Testing 

Python  scripts  using  open  source  packages  for  Natural  Language  Processing  was  the  tool  chosen 
for  analyzing  the  data  collected.  A  script  that  identifies  the  important  concepts  (words)  of  each 
sentence  in  the  minutes  resulting  from  the  engagements  allowed  us  to  identify  a  total  of  139 
research  objectives  across  the  24  engagements.  A  sense  for  the  emphasis  within  these 
identified  research  objectives,  down  to  the  resolution  of  the  sub-categories  identified  above  are 
reflected  in  Figure  1  (A  through  D).  The  data  collected  from  the  survey  of  the  INCOSE  Fellows 
was  analyzed  separately  and  is  included  in  this  report  also. 
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Management,  Culture  and  Agility 


Type 

■  Agile  Eng  neerlng 

■  Mission  Engineering 


Figure  IB.  Breakout  of  Cluster  2 


MBSE,  Decision  Making  and  Integration 


Type 

Computational  Applications  in  SE 

■  Decision  Making 

■  Model  Based  SE 

■  System  Integration 

■  Systems  Engineering  Modeling  and  Measure 


Figure  1C.  Breakout  of  Cluster  3 
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Security,  Trust,  Risk  and  Testing 


Typ# 

■  Cybersetu' ty 

■  Risk  and  Uncertainty 

■  Security,  Reliability  and  Resilience 

■  System  Trust 

■  Testing 


Figure  ID.  Breakout  of  Cluster  4 
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Research  Task  Context  and  Preparation 


Research  Task  Background 

The  Department  of  Defense  (DoD)  determines  its  research  needs  for  the  SERC  in  a  variety  of 
ways.  The  SERC  executes  the  core-funded  research  outlined  in  the  SERC  Technical  Plan,  which 
was  developed  based  on  inputs  from  the  Government's  Executive  Advisory  Board  and  the 
SERC's  Research  Council.  DoD  organizations  sponsor  their  own  tasks  within  the  SERC.  Finally, 
the  SERC  periodically  puts  out  a  call  for  incubator  tasks,  where  principal  investigators  put 
together  a  short  research  proposal  which  receives  a  modest  amount  of  funding  to  develop  a  set 
of  research  findings  and  challenges,  which  may  receive  substantial  funding  on  subsequent  task 
orders.  Although  these  avenues  and  this  information  is  helpful  in  shaping  DoD's  SE  research 
priorities,  this  particular  research  task  was  focused  on  engaging  with  S&T  and  Engineering 
leaders  across  the  DoD's  laboratories  and  engineering  centers  to  understand  if  it  was  possible 
to  identify  discernable  patterns  across  the  board  with  regard  to  research  priorities  and 
opportunities  for  impact.  This  information  would  be  useful  in  the  ongoing  initiative  to  update 
the  SERC  Technical  Plan  for  the  next  five  years. 

Research  Task  Approach 

This  research  task  was  established  to  identify  patterns  of  research  that  are  of  the  greatest 
relevance  to  the  engineering  and  technical  leaders  at  warfare  centers  in  the  Army,  Navy  and  Air 
Force.  Researchers  engaged  other  Defense  communities,  such  as  the  Intelligence  Community 
with  the  same  objective.  The  research  patterns  identified  in  this  task  will  help  guide  SE 
research  priorities  over  the  next  two  to  five  years  and  increase  the  return  on  investment  of 
future  RTs.  The  SERC  researchers  assessed  the  research  needs  through  visits  and  discussions 
with  technical  leaders  at  the  various  warfare  centers,  with  a  focus  on  the  leveraging  the  three 
core  competencies  of  the  SERC: 

1.  Long-term,  comprehensive  systems  engineering  focused  on  DoD  acquisition, 

2.  Leverage  developments  in  systems  architecting,  complex  systems  theory,  systems  thinking, 
systems  science,  knowledge  management  and  software  engineering  to  perform  research  to 
advance  the  design  and  development  of  complex  systems  across  all  DoD  and  Intelligence 
Community  domains. 

3.  Leverage  developments  in  open  systems  standards,  organizational  theory,  program 
management,  SE  management,  and  information  technology  to  provide  needed  integration 
of  program/technical  management  MPTs. 

The  research  team  also  collected  and  assessed  any  development  issues  related  to  the 
engineering  workforce. 

The  data  collection  exercise  consisted  of  24  site  visits  and  telephone  discussions.  Additionally, 
we  collaborated  with  INCOSE  to  run  a  supporting  survey  on  this  topic  to  all  the  INCOSE  Fellows 
-  worldwide  (approximately  30%  of  all  Fellows  responded  to  the  survey).  Here  is  the  list  of 
these  24  engagements: 
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1.  Sandia  National  Laboratories  -  Engagement  1 

2.  Sandia  National  Laboratories  -  Engagement  2 

3.  Jet  Propulsion  Laboratory 

4.  NSWC  -  Corona  Division 

5.  NSWC  -  Philadelphia  Division 

6.  Wright  Patterson  AFB  -  AFLCMC 

7.  Naval  Air  Weapons  Station  -  China  Lake 

8.  AMSAA-  Aberdeen  Proving  Grounds 

9.  Embedded  Systems  Institute  -  Netherlands 

10.  The  Aerospace  Corporation 

11.  The  MITRE  Corporation 

12.  AFOTEC- Kirkland  AFB 

13.  NSWC  -  Carderock 

14.  NSWC-NUWC 

15.  NSWC  -  Dahlgren 

16.  NAVSEA  Headquarters 

17.  1C  Engagement  1 

18.  1C  Engagement  2 

19.  INCOSE  Fellows  -  Worldwide  Survey 

20.  ARDEC  -  Picatinny  Arsenal 

21.  NAVSEA  -  SPAWAR 

22.  NAVSEA  05T 

23.  PEO  -  Aviation 

24.  AMRDEC  -  NASA  (Marshall)  -  Joint  Visit 
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The  site  visits  got  triggered  by  a  visit  request  and  an  attached  Terms  of  Reference  for  the  visit. 
An  exemplar  visit  request  letter  and  the  attached  Terms  of  Reference  are  included  in  Appendix 
A  of  this  report.  A  set  of  the  minutes  resulting  from  these  visits  is  included  in  Appendix  B  of  this 
report.  The  names  of  the  individuals  who  hosted  the  study  team  and  were  engaged  in 
discussions  during  the  visit  have  been  removed. 

Overview  and  Analysis  of  the  Data  Collected 


The  data  collection  exercise  consisted  of  24  site  visits  and  telephone  discussions.  Additionally, 
we  collaborated  with  INCOSE  to  run  a  supporting  survey  on  this  topic  to  all  the  INCOSE  Fellows 
-  worldwide  (approximately  30%  of  all  Fellows  responded  to  the  survey). 

Python  scripts  using  open  source  packages  for  Natural  Language  Processing  was  the  tool  chosen 
for  analyzing  the  data  collected.  A  script  that  identifies  the  important  concepts  (words)  of  each 
sentence  in  the  interviews  allowed  us  to  identified  139  total  research  objectives  across  the  24 
engagements.  The  number  of  objectives  for  each  research  center  identified  can  be  seen  in 
Figure  2.  It  is  important  to  note  that  the  numbers  disparity  is  a  reflection  of  the  nature 
(sometimes  unique,  such  as  in  test  organizations)  of  the  organization  as  well  as  whether  or  not 
they  need  to  or  are  already  covering  the  research  topic. 


Research  Center 

20 


Aerosp  AFOT..  AMRD  AMSAAARDEC  ESI  iCDoDICDoD  JPL  MITRE  NAVS..  NAVSE  NAWC  NSWC  NSWC  NSWC  NSWC  NUWC  PEO-A  SNL-1  SNl-2  SPAW..  WP 

aceC.  ECyN.  TNO  -1  -2  Corpo..  A-2  Carde  .  Corona  Dahlg.  Phila..  vlaticn  AFB 


Figure  2.  Snapshot  of  Data  Collected  across  the  various  Engagements 

Thereafter  a  secondary  script  was  developed  to  order  the  objectives  identified  based  on  their 
similarity,  creating  clusters  of  closely  related  objectives.  After  this  automated  identification  of 
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clusters,  we  manually  identified  whether  the  clusters  did  identify  human  interpretable  concepts 
and  found  clear  patterns  in  the  results.  The  objectives  were  then  divided  into  4  main  groups  of 
research  objectives,  and  these  4  groups  were  then  sub-divided  into  lower  level  objectives  that 
are  very  closely  related  to  each  other. 

The  4  primary  research  objectives  are: 

5.  Knowledge,  Data  and  Machine  Learning 

6.  Systems  Engineering  MBSE,  Decision  Making  and  Integration. 

7.  Management,  Culture  and  Agility 

8.  Security,  Trust,  Risk  and  Testing 

The  distribution  of  the  research  objectives  identified  in  this  study  across  the  4  primary 
clusters/groups  can  be  seen  in  Figure  3. 


Group 

SO 


46 


Knowledge,  Data  and  Machine  Management,  Culture  and  Agility  MBSE.  Decision  Making  and  Security,  Trust,  Risk  and  Testing 
Learning  Integration 

Figure  3.  Distribution  of  Research  Objectives  across  the  4  Primary  Clusters 

The  types  of  research  objectives  that  these  groups  were  divided  in  are  described  in  the  sections 
that  correspond  to  each  group.  An  important  part  of  the  data  is  that  the  interest  of  the 
research  centers  varies  significantly  between  groups  of  research  objectives.  The  only  research 
center  that  approached  a  uniform  distribution  of  interests  was  Aerospace  Corp,  while  other 
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research  centers  such  as  AFOTEC  and  ESI  TNO  had  only  interests  in  one  or  two  of  the  groups. 
The  distribution  of  interests  of  the  across  the  engagement  sources  is  shown  in  Figure  4. 


Thereafter,  the  following  sections  show  the  distribution  of  identified  research  interests  and 
research  centers  in  each  of  the  four  groups.  First,  each  section  presents  the  objectives 
organized  in  the  identified  cluster.  Afterwards  for  each  section  the  following  things  are 
presented: 

•  A  first  bar  or  pie  chart  describing  the  amount  of  research  interests  that  each  research 
center  has  in  that  group. 

•  A  packed  bubble  chart  describing  the  amount  of  research  interests  in  each  research 
type  (the  sub  clusters  that  each  research  group  is  contained).  This  chart  also 
describes  the  amount  of  research  interests  that  each  research  center  identified  in 
each  type. 

•  A  tree-map  analyzing  the  patterns  of  the  interests  and  the  related  distribution  among 
the  research  centers. 


Research  Center 


Group 

|  Knowledge,  Data  and  Machine  Learning 
H  Management,  Culture  and  Agility 
H  MDSE .  Dec  sion  Making  and  Integration 
|  Security.  Trust,  Risk  and  Testing 


Figure  4.  Distribution  of  Research  Interests  across  the  Data  Sources 


Cluster  1:  Knowledge,  Data  and  Machine  Learning 


The  first  cluster  identified  was  called  Knowledge ,  Data  and  Machine  Learning.  It  contains  all  the 
identified  research  interests  that  are  data  centric,  either  dealing  with  the  management  and 
complexity  of  the  data  or  the  application  of  data  manipulation  techniques  such  as  Al/Machine 
Learning  and  Data  Analytics.  The  primary  sub-categories  in  this  cluster  area  include  the 
following: 
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•  Al/Machine  Learning 

•  Data  Analytics 

•  Data/Complexity 

•  Knowledge  Management  and  Document/Requirements 

The  two  centers  with  the  biggest  amount  of  discussed  research  interests  in  this  area  were 
Sandia  National  Laboratory  and  JPL.  Particularly  important  patterns  identified  include: 

•  JPL  discussed  the  widest  interest  in  knowledge,  document  and  requirement 
management.  They  showed  a  significant  interest  in  concepts  such  as  language 
regularization  and  ambiguity  minimization. 

•  The  other  types  of  research  projects  had  a  very  diverse  and  uniform  distribution  of 
research  centers  interested,  showing  the  prevalence  of  data  size,  availability  and 
complexity  issues  that  have  arisen.  This  is  perfectly  coherent  with  the  big  surge  of  the 
interest  in  the  (arguably)  buzzword  big  data,  which  mainly  deals  with  the  massive 
amounts  of  information  being  produced. 

•  The  Data/Complexity  type  of  research  interest  was  of  greatest  interest  among  all  the 
types  of  research  types  in  all  the  research  groups,  once  again  reinforcing  the 
importance  of  dealing  with  big  amounts  of  data  and  showing  this  as  one  of  the  main 
areas  of  research  identified  in  the  set  of  interviews. 

•  Even  though  smaller  in  the  amount  of  research  objectives  contained,  if  we  combine 
Al/Machine  Learning  and  Data  Analytics,  which  are  disciplines  with  a  big  intersection, 
they  also  represent  an  important  area  of  research  identified.  AFOTEC  (test  focus 
organization)  was  the  main  research  center  interested,  but  overall  these  research 
areas  had  a  big  uniformity  and  number  of  research  centers  with  interests  in  them. 

•  The  overall  uniformity  in  the  distribution  of  the  research  types  Data/Complexity, 
Al/Machine  Learning  and  Data  Analytics  show  that  projects  in  these  areas  have  a  very 
big  potential  of  collaborating  and/or  impacting  multiple  research  centers  at  the  same 
time. 
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Number  of  Records 


Aerospace  AFOTEC  ESI  TNO  iCDcO-1  ICDoD-2  JPL  MITRE  Co  NAVSEA  NAVSEA-  NSWC  NSWC  NSWC  SNl-1  SNl-2  SPAWAR  WPAFB 

Corp  rporaticn  2  Carderock  Corona  Dahlgreen 
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Type 

H  Al/Machine  Learning 
|  Data  Analytics 
|  Data/Complexity 

H  Knowledge  Management  and  Document/Requirements 


Report  No.  SERC-2018-1R-102 


12 


31  January  2018 


Data/Complexity 

ESITNO 

Knowledge  Management  and 
Document/Requirements 

JPL 

Data/Complexity 

NSWC  Corona 

Data/Complex  ty 
MITRE  Corporation 

Data/Complexity 

WPAFB 

Data/Complex  ty 

SNL-1 

Knowledge 

Management  and 

Al/Machine  Learning 

AFOTEC 

Al/Machine  Learning 
JPL 

Al/Machine  Learning 
NSWC  Corona 

Data  Data 

Analytics  Analytics 

AFOTEC  NAVSEA -2 


Data  Data 

Analytics  Analytics 

NSWC  Corona  NSWC 

Dahlgreen 


Data  Data 

Analytics  Analytics 

SNl-1  SNl-2 


Al/Machine  Learning 
NAVSEA -  2 


Al/Machine  Learning  Al/Machine  Learning 
NAVSEA  SNL-1 

Knowledge  Management  and 
Document/Requiremerts 
NSWC  Carderock 


Data  Analytics 
WPAFB 


Cluster  2:  MBSE,  Decision  Making  and  Integration 


The  second  cluster  of  research  interest  identified  was  identified  as  Systems  Engineering:  MBSE, 
Decision  Making  and  Integration.  Most  research  topics  identified  came  from  SPAWAR  and  DoD 
1C.  Even  though  these  were  clearly  the  leading  centers  for  number  of  identified  research 
interests,  there  was  still  a  big  number  of  other  research  centers  and  uniformity  in  this  cluster. 
The  primary  sub-categories  in  this  cluster  include: 

•  Computational  Applications  in  SE 

•  Decision  Making 

•  Model  Based  SE 

•  System  Integration 

•  Systems  Engineering  Modeling  and  Measurement 

Particularly  important  patterns  identified  include: 

•  System  Integration  was  the  type  of  research  that  had  the  biggest  amount  of  research 
interests  in  it.  1C  DoD,  SPAWAR  and  Aerospace  Corp  were  the  research  centers  that 
identified  the  most  interested  in  it,  but  overall  it  contained  a  big  number  of  research 
centers  with  interest  in  it.  Particularly  notable  is  that,  even  though  this  type  of 
research  is  identified  in  this  second  cluster  of  interest  related  to  Systems  Engineering, 
it  arguably  is  closely  related  to  the  research  type  of  Data/Complexity  in  the  first 
cluster  (Knowledge,  Data  and  Machine  Learning).  This  is  coherent  with  the  analysis 
presented  in  that  section,  since  it  is  not  hard  to  argue  that  at  least  theoretically  an 
increase  in  Data/Complexity  would  have  a  positive  feedback  loop  with  an  interest  in 
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System  Integration  type  of  research. 

•  Model  Based  SE  was  one  of  the  main  identified  interests  of  NUWC,  being  responsible 
of  a  considerable  percentage  of  the  research  interests  in  this  area.  Some  of  the 
research  interests  contained  in  it  were  closely  related  with  other  areas,  including 
concepts  such  as  model  testing  and  model  verification. 

•  There  was  a  big  interest  in  the  area  of  general  Systems  Engineering  Modeling  and 
Measure,  which  includes  overall  quantification  and  effectiveness  measures  of 
Systems  Engineering  concepts.  The  center  with  the  highest  discussed  interest  in  this 
area  was  SPAWAR. 

•  There  is  a  wide  interest  in  research  on  Verification  and  Validation.  NUWC  Newport 
mentioned  model  validation  in  general  and  MBSE  validation  in  particular.  NSWC 
Corona  asked  for  help  with  model  verification  in  the  presence  of  small  data  sets,  and 
AMSAA  asked  about  better  understanding  of  V&V  and  accreditation  of  models  and 
analysis  tools.  JPL  mentioned  V&V  for  learning  systems,  Sandia  brought  up  V&V  for 
additive  manufacturing,  and  AMRDEC  emphasized  the  issue  of  validation  within 
systems  engineering  in  general. 

•  One  interesting  area  identified  was  the  Applications  of  Computer  Science  Concepts. 
Even  though  small,  the  research  interests  identified  represented  a  different 
combination  of  disciplines  than  most  of  the  research  interests  identified.  Discussed 
mainly  by  1C  DoD,  it  consisted  of  trying  to  apply  Computer  Science  concepts  and 
disciplines  to  Systems  Engineering. 
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SPAWAR 


Type 

|  Computational  Applications  in  SE 
Decision  Making 
|  Model  Based  SE 
System  Integration 

|  Systems  Engineering  Modeling  and  Measure 
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Model  Based  SE 

NUWC 

Model  Based  SE 

AROCC 

Mode  Based  SE 
AMROEC/NASA 
MSEC 

Mode:  Based  SE 

MITRE 

Corporation 

Model  Based  SE 
NAVSEA-2 

System  integration 

SPAWAR 

System  System 

Integration  Integration 

1C  DoD *  *2  JPl 

Model  Based  SE 

1C  DoD-1 

Mode:  Qased  SE 

KAWC 

Model 

Based  SE 
SPAWAR 

System  integration 

V/PAF8 

Model  Based  SE 

NSWC  Cardcrock 

Mode:  Based  SE 

System  integration 

Aerospace  Corp 

Model  Based  SE 

PEO  -  Aviation 

NSWC  Corona 

Computational  Applications 
InSE 

Decision  Making 

Aerospace  Corp 

Mode:  Based  SE 

SMI-1 

1C  DoD-2 

Systems  Engineering  Modeling  ary]  Measure 


Systems  Engineering  Modolingand  Measure 
PEO  -  Aviation 


Systems  Engineering 
Modeling  and  Measure 
AMSAA 


Systems  Engineering 
Modeling  and  Measure 
ICDoO-Z 


Systems  Engineering 
Modeling  and  Measure 
KSWC  Corona 


Systems  Engineering 
Modeling  and  Measure 
NSWC  Philadelphia 


Computational  Applications  in  SE 
NAVSCA-2 


Computational  Applications  in  SE 
WPAFB 


Decision  Making 
SNl  1 


Decision  Malcng 
WPAEB 


Cluster  3:  Mission  Engineering  and  Agility 


The  third  cluster  of  research  interests  identified  was  called  Mission  Engineering  and  Agility.  Not 
only  is  this  research  cluster  different  in  its  nature  that  the  others,  it  is  also  significantly  different 
in  its  distribution: 

•  Unlike  all  the  other  clusters,  one  research  center  identified  the  greater  amount  of 
research  interests  identified  in  this  cluster:  DoD  1C.  Combining  the  interests  in  both 
of  its  engagements,  DoD  1C  is  source  for  over  identification  of  half  of  the  research 
interests  in  this  cluster. 

Overall  the  "dominance"  of  one  research  center  makes  this  cluster  of  research  interests 
significantly  different  than  the  others,  and  is  a  clear  consequence  of  the  focus  of  1C  DoD  of 
dealing  with  training  Systems  Engineers.  Overall,  pursuing  a  research  interest  in  this  area  would 
be  highly  conductive  with  collaboration  and/or  impact  with  1C  DoD  mainly,  with  some 
secondary  research  centers  like  Aerospace  Corp. 

The  primary  sub-categories  in  this  cluster  include: 

•  Mission  Engineering 

•  Agile  Engineering 
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SPAWAR 


Type 

Agile  Engineering 
|  Mission  Engineering 
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Miss- on  Engineering 
ICDoO-2 


Agile  Engineering 
1C  DoD-2 


Agile  Enfl-neering 
SNL-1 


Mission 
Eng  veering 
NSWC  PniiadelpfM 


Agile  Engineering 
NAWC 


Ag'le  Engmeer-ng 
NSWC  Pn.ladeipnia 


Agile  Engineering  Agile  Engineering 

NSWC  Carder ot<  NSWC  Corona 


Mission  Engineering 
MITRE  Corporation 


Miss>on  Engineering  Mission  Engineering 

NSWC  Carderock  NUWC 


Agile  Engineering  Agile  Engineering  Agile  Engineering 

AMSAA  NUWC  SPAWAR 


Mission 

Engineering 

WPAFB 


Agile  En^neering 
ARDCC 


Mission  Engineering 
NAVSEA 

Agile  En^netring  Agile  Engineering 

Aerospace  Coro  WPAFB 


Mission  Engineering 
SPAWAR 


Cluster  4:  Security,  Trust,  Risk  and  Testing 


The  last  research  interest  cluster  identified  was  called  Security,  Trust,  Risk  and  Testing.  Given  its 
nature  it  is  related  in  a  considerable  manner  to  the  second  cluster,  though  given  the  clear 
distinction  and  amount  of  research  interests  it  was  deemed  appropriate  to  create  the  fourth 
cluster  separated  from  the  second  one. 

This  research  cluster  had  no  single  research  center  as  the  most  interested  in  it,  but  had  a  high 
uniformity  in  the  distribution  of  research  centers  interested.  The  primary  sub-categories  in  this 
cluster  include: 

•  Cyber-security 

•  Risk  and  Uncertainty 

•  Security,  Reliability  and  Resilience 

•  System  Trust 

•  System  Testing 

Particularly  important  patterns  identified  include: 

•  System  Trust  was  the  biggest  research  type  identified,  with  almost  40%  of  the  research 
interests  in  the  cluster.  This  type  of  research  contained  interest  of  a  large  number  of 
research  centers,  which  mainly  dealt  with  the  difficulty  of  achieving  concepts  such  as: 
system  trust,  source  of  truth,  system  definition  and  validation.  It  is  very  closely  related  to 
the  Risk  and  Uncertainty  and  the  Security,  Reliability  and  Resilience  types  of  research. 
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Number  of  Records 


and  if  those  three  are  counted  together  they  represent  over  66%  of  the  interests  in  this 
cluster,  with  11  research  centers  interested  in  this  areas. 

•  The  Testing  area  was  a  big  priority  for  AFOTEC  (as  would  be  expected),  with  this 
research  center  accounting  for  all  minus  one  of  the  research  interests  in  this  group. 

•  Cyber  security  was  mainly  identified  interests  of  NSWC  Corona,  with  NAVSEA  and  NSWC 
Dahlgren  also  expressing  explicit  interest  in  this  type  of  research,  it  could  be  an  area 
that  rises  as  sub  interests  of  some  of  the  System  Security  interests  in  System  Trust,  so  it 
could  be  deceptively  small.  Overall  there  was  a  big  number  of  research  centers 
interested  in  this  area,  similar  to  the  first  cluster  projects  in  this  areas  have  a  very  big 
potential  of  collaborating  and/or  impacting  multiple  research  centers  at  the  same  time. 
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r 

1 

1  NS 

Testing 
iWC  Dahlgre. 

e^i 

Type 

|  Cybersecurity 
|  Risk  and  Uncertainty 
|  Security,  Reliability  and  Resilience 
System  Trust 
|  Testing 
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System  Trust 

System  Trust 

System  Trust 

System  Trust 

AMSAA 

1C  DoD  1 

NSWC  Corona 

NSWC  Dahlgreen 

System  Trust 

JPl 

System  Trust 

System  Trust 

NUWC 

SNl-1 

System  Trust 

NSWC  Carderock 

System  Trust 

Security.  Reliability  and  Res<  Hence  Security, 

WPAFB  Reliability  and 

Resilience 
AMRDEC/NASA 
MSFC 


Security,  Security,  Security, 

Reliability  and  Reliability  and  Reliability  and 

Resilience  Resilience  Resilience 

Aerospace  Corp  iCDoD-2  SNL-2 


Risk  and  Uncertainty 

Risk  and  Uncertainty 

Risk  and  Risk  and 

AMRDEC/N ASA  MSFC 

JPL 

Uncertainty  Uncertainty 

NSWC  SNl  •  1 

Philadelphia 

Risk  and  Uncertainty 

Risk  and  Uncertainty 

AMSAA 

NSWC  Carder ock 

Risk  and  Uncertainty 

Risk  and  Uncertainty 

Risk  and  Uncertainty 

Aerospace  Corp 

NSWC  Corona 

SPAWAR 

Security.  Reliability  and  Resi  lienee 
SPAWAR 


Cybcrsccurlty 
NSWC  Corona 


Cybcrsccurlty 

NAVSEA 


Cyber  security 
NSWC  Dahlgreen 


Workforce  Issues 


Workforce  issues  were  discussed  explicitly  in  most  of  the  24  engagements.  Some  of  the 
significant  concerns  included  the  following: 

•  It  is  often  hard  to  compete  against  industry  for  talent,  particularly  IT/Software 
developers. 

•  In  the  current  environment  of  competition  for  cybersecurity  talent,  some 
organizations  are  falling  back  on  training  to  develop  cybersecurity  expertise,  as 
opposed  to  hiring  experts.  There  is  a  perceived  need  for  DoD  coursework  to  support 
this  strategy.  This  need  was  mentioned  by  NSWC  Dahlgren  and  NUWC  Newport. 

•  Upcoming  retirement  of  a  significant  portion  of  the  workforce  and  the 
commensurate  loss  of  domain  knowledge  and  experience.  This  notion  of  loss  of 
domain  knowledge  was  brought  up  during  almost  all  our  engagements. 

•  Lack  of  trust  among  major  organizations,  particularly  between  acquisition  and 
contractors,  inhibits  capability. 

•  Concern  about  transitioning  to  digital  engineering  due  to  a  lack  of  talent  and 
capability  at  all  levels  of  seniority. 

•  Concern  about  skills  relating  to  software  and  systems  engineering  and  architecting; 
system  security;  analytics  and  machine  learning.  This  is  particularly  true  when  hiring 
individuals  who  have  the  ability  to  go  through  the  clearance  process. 

The  topic  of  replacement  of  lost  expertise  due  to  retirement  or  other  attrition  led  to  some 
discussion  pertaining  to  mentoring  needs  and  enhanced  training  of  the  entry  level  workforce. 
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Survey  Results:  INCOSE  Fellows 


This  section  reflects  the  input  received  from  the  survey  of  the  INCOSE  Fellows,  worldwide. 


Model  Based  SE 

System  Integration 

20 

7 

Computational 

Decision 

Applications  tnSE 

Making 

5 

3 

Systems  Engineering  Modeling  and  Measure 

11 

The  general  Systems  Engineering  Modeling  and  Measures  along  with  research  priorities  in 
Model  Based  SE  were  by  far  the  most  common  ones  for  the  fellows,  occupying  75%  of  the 
interests  of  Systems  Engineering. 
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Conclusion 


This  project  has  provided  a  strong  demand  signal  with  regard  to  the  research  priorities  in  the 
DoD  ecosystem,  from  the  perspective  of  technical  leaders.  The  five  broad  areas  of  research 
emphasis,  based  on  the  data  collected  area: 


1.  Multiple  topics  relating  to  Mission  Engineering 

2.  Multiple  topics  relating  to  Agile  Development  and  Engineering  at  the  Scale  of  the 
Enterprise 

3.  Model  Based  Systems  Engineering 

4.  Modeling  and  assessing  System  Trust,  Security,  and  Resilience 

5.  Development  of  Pragmatic  Methods  to  deal  with  and  to  leverage  increasing  Complexity 
and  Analytics,  and  leveraging  machine  learning  for  this  purpose. 


Group 

■  Knowledge.  Data  and  Machine  Learning 

■  Management,  Cultu'e  and  Agility 

■  MBSE,  Decision  Making  and  Integration 

■  Security,  Trust.  Risk  and  Testing 
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Appendix  A:  Example  Visit  Request  and  the  attached  Terms  of  Reference 


Name  and  Title 

Organization 
Address  1 
Address  2 

Subject:  Systems  Engineering  Research  Needs  and  Workforce  Development  Study 
Dear  XYZ: 

The  DASD,  Systems  Engineering,  within  the  Office  of  the  Secretary  of  Defense  has  created  and  funded  a 
University  Affiliated  Research  Center  (UARC)  to  focus  on  systems  engineering:  the  Systems 
Engineering  Research  Center  (SERC).  It  is  critical  to  supplement  SERC  research  priorities  with  input 
from  technical  leaders  at  the  various  warfare  centers  within  the  Army,  Navy,  Air  Force,  and  Marine 
Corps,  given  the  SERC  charter  to  conduct  research  leading  to  improved  methods,  tools,  and  processes  to 
improve  the  practice  of  systems  engineering  on  complex,  complicated  programs  within  the  Department  of 
Defense.  This  research  task  is  focused  on  understanding  the  current  state  of  engineering  practice  and  the 
potential  gaps  that  should  be  addressed  by  research. 

This  study  will  systemically  address  the  following  questions: 

1 .  How  do  you  apply  systems  engineering  to  complex  programs?  How  do  you  address  system  and 
program  resiliency  and  flexibility?  What  is  the  role  of  SE  in  analyzing  needs  and  war-fighting 
capabilities,  simulations/gaming,  concept  analyses,  forming  of  CONOPS,  program  formulation 
and  execution,  test  and  integration,  sustainment  and  capability  evolution,  and  training? 

2.  What  is  your  organizational  policy  on  systems  engineering?  How  do  you  assess  program  and 
system  quality  attributes?  Are  your  workforce  needs  being  met?  Can  you  share  with  us  your 
current  state  of  practice  with  regard  to  engineering  tools  and  environments? 

3.  From  your  point  of  view,  what  are  the  shortfalls,  and  key  engineering  and  systems  engineering 
challenges  that  should  be  addressed  through  focused  research,  leading  to  the  development  of  new 
enabling  methods,  tools,  and  capabilities? 

Our  approach  to  these  discussions  will  be  very  broad  based  to  understand  all  engineering,  technology,  and 
workforce  related  matters  of  concern  to  the  technical  leaders  at  visited  centers.  Focus  will  be  on  support 
of  the  warfighter,  particularly  through  rapid  delivery  of  effective  technology. 

We  would  like  to  have  several  members,  potentially  including  Ms.  Baldwin  (DASD  (SE)  -  Acting),  of 
our  study  group  visit  NAVSEA  sometime  during  the  week  of  October  16,  subject  to  personnel 
availability. 

During  our  visit  we  would  like  to  be  briefed  on  the  overall  NAVSEA  mission,  along  with  your  current 
capabilities  in  engineering  and  systems  engineering  as  discussed  above.  As  part  of  this  briefing,  we  would 
welcome  your  thoughts  on  engineering  and  systems  engineering  capability  gaps  that  should  be  addressed 
via  research,  together  with  your  thoughts  on  the  development  of  the  engineering  workforce  today  and  into 
the  future.  We  would  be  able  to  provide  you  with  a  briefing  on  the  Systems  Engineering  Research  Center 
and  its  current  research  portfolio.  Please  suggest  potential  dates  for  our  visit  and  provide  a  draft  agenda  at 
your  earliest  convenience.  Supporting  descriptive  material  that  you  might  choose  to  provide  in  advance 
would  be  helpful. 
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Thank  you  for  your  assistance  in  this  important  endeavor.  We  look  forward  to  our  visit  and  to  learning 
about  the  comprehensive  range  of  NAVSEA  activities. 


Sincerely, 


Dinesh  Verma,  Ph.D. 
Executive  Director,  SERC 


Copy: 

Ms.  Kristen  Baldwin,  DASD  (Systems  Engineering),  Acting 

Mr.  Scott  Lucero,  Deputy  Director,  Strategic  Initiatives,  ODASD  (Systems  Engineering) 

Attachment: 

Systems  Engineering  Research  Needs  and  Workforce  Development  Assessment  Study  TOR 

Terms  of  Reference:  Systems  Engineering  Research  Needs  and  Workforce  Development 
Assessment  Study  (SERC  RT-174) 

Background 

The  DoD  makes  a  significant  investment  in  research  intended  to  improve  systems  engineering.  The 
direction  for  this  research  is  determined  by  responsible  OSD  personnel  advised  by  an  executive  advisory 
board  government  experts,  as  well  as  industry  experts,  and  active  systems  engineering  researchers.  This 
approach  has  been  productive,  but  to  develop  a  research  agenda  that  directly  addresses  critical  future 
warfighter  needs,  systems  engineering  shortfalls  and  opportunities  and  also  future  engineering  workforce 
needs,  research  priorities  must  be  understood  from  the  point  of  view  of  DoD’s  laboratories  and 
engineering  centers. 

Context  and  Rationale 

The  DASD  for  Systems  Engineering,  within  the  Office  of  the  Secretary  of  Defense,  has  created  and 
funded  a  University  Affiliated  Research  Center  (UARC)  to  focus  on  systems  engineering  research:  the 
Systems  Engineering  Research  Center  (SERC).  It  is  critical  to  supplement  SERC  research  priorities  with 
input  from  technical  leaders  at  the  various  warfare  centers  within  the  Army,  Navy,  Air  Force,  and  Marine 
Corps,  given  the  SERC  charter  to  conduct  research  leading  to  improved  methods,  tools,  and  processes  to 
improve  engineering  on  complex  and  complicated  programs  in  the  Department  of  Defense.  This  research 
task  will  be  focused  on  reducing  the  air  gap  between  such  programs  and  the  research  being  conducted  by 
the  faculty  at  the  various  SERC  collaborator  universities.  The  objective  of  this  study  will  be  to  understand 
the  specific  systems  engineering  research  and  workforce  needs  at  the  various  warfare  centers,  and  to 
identify  the  common  themes  that  extend  across  different  DoD  organizations. 

Approach  and  Objectives 

In  close  coordination  with  the  DASD  (SE),  the  research  team  will  schedule  visits  to  the  various  Army, 
Navy,  Air  Force,  and  Marine  Corps  warfare  centers.  They  will  also  plan  visits  to  MDA  and  selected  IC 
organizations.  Illustrative  centers  include: 
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•  NSWC,  NUWC  and  NAVAIR  -  Various  divisions 

•  SPA  WAR 

•  SMC 

•  Army  RDECs 

•  Army  COE  ERDC 

•  Army  TRADOC,  especially  ARCIC 

•  AF  SMC,  ACC,  and  AFLCMC 

•  AFRL,  NRL,  ARL 

•  Representative  PEOs 

•  Joint  organizations  such  as  MDA  and  USSOCOM 

•  Relevant  FFRDCs  and  UARCs 

The  approach  to  these  discussions  will  be  broad-based  to  understand  all  engineering,  technology,  and 
workforce  related  matters  of  concern  to  the  technical  leaders  at  these  centers.  Focus  will  be  on  support  of 
the  warfighter,  particularly  through  rapid  delivery  of  effective  technology. 

Each  visit  will  be  memorialized  with  a  visit  report  and  white  paper  outlining  the  attendees,  the  agenda, 
any  tours  or  lab  visits,  and  key  takeaways  from  the  discussions  regarding  the  pain  points  with  regard  to 
workforce  related  matters,  research  and  development  related  matters,  and  related  risks. 

The  various  visit  reports  and  white  papers  will  then  be  consolidated  into  a  final  report  and  an  executive 
presentation,  identifying  SE  shortfalls  and  opportunities  for  research  across  the  DoD  engineering  and 
warfare  centers.  The  research  patterns  identified  in  this  task  will  help  guide  systems  engineering  research 
priorities  over  the  next  two  to  five  years  and  increase  the  return  on  investment  of  future  SERC  research 
tasks. 

Deliverables 

A008  Status  report  (bi-monthly) 

A009  Technical  and  Management  Work-plan  (due  at  the  start  of  the  project) 

AO  13  Final  Technical  Report  (due  at  the  completion  date) 

Stakeholders,  Roles  and  Responsibilities,  and  the  Format  of  the  Study 

The  study  will  be  conducted  by  the  Core  Study  Team,  with  guidance  and  review  by  the  Study  Steering 
Team.  Team  members  are  delineated  in  Table  1 . 

Table  1:  Study  Team 


Core  Study  Team: 

Dr.  Dinesh  Verma,  SERC  (Stevens  Institute  of  Technology)  - 
Chair  of  the  Core  Study  Team 

Dr.  Paul  Collopy,  SERC  (University  of  Alabama  -  Huntsville) 

Dr.  Spiros  Pallas,  SERC  (Stevens  Institute  of  Technology) 

Study  Steering  Team: 

Ms.  Kristen  Baldwin,  DASD  (Systems  Engineering),  Acting 

Mr.  D.  Scott  Lucero,  Deputy  Director,  Strategic  Initiatives, 
ODASD  (Systems  Engineering) 
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Appendix  B:  Listing  of  All  Research  Interests  from  Engagement  Minutes 


Research  Objective 

Type 

Group 

Research 

Center 

Knowledge  Management 

Knowledge  Management  and 
Document/Requirements 

Knowledge,  Data 
and  Machine 

Learning 

JPL 

capture  institutional  knowledge 

Knowledge  Management  and 
Document/Requirements 

Knowledge,  Data 
and  Machine 

Learning 

SNL  -  1 

paper  based  process,  need  consistent  in 
information  continuity  and  consistency. 

Knowledge  Management  and 
Document/Requirements 

Knowledge,  Data 
and  Machine 

Learning 

SNL  -  1 

rigorous  basis  to  underpin  the  information 
flows  to  prevent  ambiguity  and  imprecision 

Knowledge  Management  and 
Document/Requirements 

Knowledge,  Data 
and  Machine 

Learning 

JPL 

System  requirements  do  not  address 
integrating  one  system  with  another. 

Knowledge  Management  and 
Document/Requirements 

Knowledge,  Data 
and  Machine 

Learning 

Aerospace 

Corp 

get  knowledge  driven  operations 

Knowledge  Management  and 
Document/Requirements 

Knowledge,  Data 
and  Machine 

Learning 

IC  DoD  -  1 

Requirements  today  are  written  at  the 
system  or  subsystem  level,  not  at  the 
mission  level. 

Knowledge  Management  and 
Document/Requirements 

Knowledge,  Data 
and  Machine 

Learning 

NUWC 

Language  Regularization 

Knowledge  Management  and 
Document/Requirements 

Knowledge,  Data 
and  Machine 

Learning 

JPL 

remove  ambiguity  from  requirements 
specifications 

Knowledge  Management  and 
Document/Requirements 

Knowledge,  Data 
and  Machine 

Learning 

JPL 

good  systems  engineering  in  environments 
where  everything  is  continually  changing 

Data/Complexity 

Knowledge,  Data 
and  Machine 

Learning 

Aerospace 

Corp 

System  Security  at  Scale 

Data/Complexity 

Knowledge,  Data 
and  Machine 

Learning 

SNL -2 
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What  performance  assessment  data  should 
be  collected  from  autonomous  systems 

Data/Complexity 

Knowledge,  Data 
and  Machine 

Learning 

NSWC 

Corona 

enhance  the  fidelity  of  condition  based 
maintenance  data 

Data/Complexity 

Knowledge,  Data 
and  Machine 

Learning 

NSWC 

Corona 

vast  amounts  of  field  data. 

Data/Complexity 

Knowledge,  Data 
and  Machine 

Learning 

AMSAA 

valuate  the  air  worthiness  of  an  increasingly 
more  complex  aviation/weapon  system 

Data/Complexity 

Knowledge,  Data 
and  Machine 

Learning 

WP  AFB 

Managing  Complexity: 

Data/Complexity 

Knowledge,  Data 
and  Machine 

Learning 

SNL  -  1 

measure  system  complexity  (with  an 
emphasis  on  software) 

Data/Complexity 

Knowledge,  Data 
and  Machine 

Learning 

WP  AFB 

Agility  at  Scale 

Data/Complexity 

Knowledge,  Data 
and  Machine 

Learning 

SNL  -  1 

Scalability 

Data/Complexity 

Knowledge,  Data 
and  Machine 

Learning 

ESI  TNO 

Performance  Assessment  is  challenged  by 
the  volume  of  data 

Data/Complexity 

Knowledge,  Data 
and  Machine 

Learning 

NSWC 

Corona 

Requirements  are  getting  out  of  hand 

Data/Complexity 

Knowledge,  Data 
and  Machine 

Learning 

JPL 

everybody  has  a  phone  and  data  - 
complexity  of  the  enterprise  is  unique 

Data/Complexity 

Knowledge,  Data 
and  Machine 

Learning 

IC  DoD  -  2 

Key  drivers  that  will  cause  (embedded) 
sytem  complexity  to  further  increase 

Data/Complexity 

Knowledge,  Data 
and  Machine 

Learning 

ESI  TNO 
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Mission  analysis  and  engineering  for 
complex  systems  and  SoS 

Data/Complexity 

Knowledge,  Data 
and  Machine 

Learning 

NSWC 

Dahlgreen 

Autonomous  Topological  Predictor- 

Corrector  Learning 

AI/Machine  Learning 

Knowledge,  Data 
and  Machine 

Learning 

AFOTEC 

Artificial  intelligence 

AI/Machine  Learning 

Knowledge,  Data 
and  Machine 

Learning 

JPL 

Machine  Learning  to  focus  on  colleting  and 
creating  test  metrics; 

AI/Machine  Learning 

Knowledge,  Data 
and  Machine 

Learning 

AFOTEC 

AI  to  support  Design: 

AI/Machine  Learning 

Knowledge,  Data 
and  Machine 

Learning 

SNL  -  1 

need  for  data  analytics  for  digital  twins 

Data  Analytics 

Knowledge,  Data 
and  Machine 

Learning 

NSWC 

Corona 

Data  analytics  -  infrastructure  learning 
tools; 

Data  Analytics 

Knowledge,  Data 
and  Machine 

Learning 

AFOTEC 

Data  Analytics  and  Support  of  Man- 
Machine  Teaming 

Data  Analytics 

Knowledge,  Data 
and  Machine 

Learning 

SNL -2 

Data  Analytics 

Data  Analytics 

Knowledge,  Data 
and  Machine 

Learning 

SNL  -  1 

Data  Analytics 

Data  Analytics 

Knowledge,  Data 
and  Machine 

Learning 

NSWC 

Dahlgreen 

system  analytics  to  develop  the  notion  of  a 
unique  system  specific  DNA 

Data  Analytics 

Knowledge,  Data 
and  Machine 

Learning 

WP  AFB 

Increasing  focus  on  model  based  testing 

Model  Based  SE 

MBSE,  Decision 
Making  and 

Integration 

SNL  -  1 

model  verification  in  the  presence  of  small 
data  sets 

Model  Based  SE 

MBSE,  Decision 
Making  and 

Integration 

NSWC 

Corona 
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how  to  verify  and  validate  a  model 
(particularly  MBSE) 

Model  Based  SE 

MBSE,  Decision 
Making  and 

Integration 

NUWC 

translating  natural  language  specifications 
and  design  documents  into  MBSE 

Model  Based  SE 

MBSE,  Decision 
Making  and 

Integration 

NUWC 

using  MBSE  to  find  flaws  in 
interoperability 

Model  Based  SE 

MBSE,  Decision 
Making  and 

Integration 

NUWC 

potential  of  Model-Based  Systems 
Engineering  (MBSE)  to  connect  disparate 
models 

Model  Based  SE 

MBSE,  Decision 
Making  and 

Integration 

NAWC 

System  requirements  do  not  address 
integrating  one  system  with  another. 

System  Integration 

MBSE,  Decision 
Making  and 

Integration 

Aerospace 

Corp 

No  Overhead  of  successful  integration,  e.g., 
55%  of  cost  if  its  not  invented  here 

System  Integration 

MBSE,  Decision 
Making  and 

Integration 

IC  DoD  -  2 

rigorous  processes  for  integrating  existing 
components  into  new  systems. 

System  Integration 

MBSE,  Decision 
Making  and 

Integration 

JPL 

case  study  conducted  on  modularizing  a 
system,  documenting  the  costs  and  benefits 
of  modularity 

System  Integration 

MBSE,  Decision 
Making  and 

Integration 

NUWC 

Certification  of  systems  that  are  being  used 
much  more  beyond  their  design  life 

System  Integration 

MBSE,  Decision 
Making  and 

Integration 

SNL  -  1 

Systems  engineering  needs  to  be  applied  at 
the  enterprise  level,  across  many  systems 

System  Integration 

MBSE,  Decision 
Making  and 

Integration 

Aerospace 

Corp 

cost  of  integration  at  the  mission  thread 
level 

System  Integration 

MBSE,  Decision 
Making  and 

Integration 

IC  DoD  -  2 

invest  in  interoperability  among  platforms 
and  systems 

System  Integration 

MBSE,  Decision 
Making  and 

Integration 

NAWC 

What  constraints  to  integrate  faster 

System  Integration 

MBSE,  Decision 
Making  and 

Integration 

IC  DoD  -  2 

communication  mechanism  to  convey  a 
sense  of  reality  as  new  technologies  evolv 

System  Integration 

MBSE,  Decision 
Making  and 

Integration 

WP  AFB 

What  is  the  career  development  path  that 
can  produce  a  system  architect? 

Systems  Engineering 

Modeling  and  Measure 

MBSE,  Decision 
Making  and 

Integration 

AMSAA 

system  framework  acknowledges 

requirements  are  not  well  known  on  the 
front  end 

Systems  Engineering 

Modeling  and  Measure 

MBSE,  Decision 
Making  and 

Integration 

NSWC 

Philadelphia 
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Mission  Engineering  seems  like  a  step  too 
far 

Systems  Engineering 

Modeling  and  Measure 

MBSE,  Decision 
Making  and 

Integration 

WP  AFB 

Quantification  of  anything  in  SE 

Systems  Engineering 

Modeling  and  Measure 

MBSE,  Decision 
Making  and 

Integration 

JPL 

effectiveness  of  SE  be  measured 

Systems  Engineering 

Modeling  and  Measure 

MBSE,  Decision 
Making  and 

Integration 

JPL 

how  do  you  characterize  the  system 
boundary 

Systems  Engineering 

Modeling  and  Measure 

MBSE,  Decision 
Making  and 

Integration 

IC  DoD  -  2 

Establish  rigor  in  systems  engineering  for 
acquisition  programs 

Systems  Engineering 

Modeling  and  Measure 

MBSE,  Decision 
Making  and 

Integration 

NSWC 

Corona 

allocate  requirements  from  the  mission  level 
to  the  system  level  to  the  sub-system  level 

Systems  Engineering 

Modeling  and  Measure 

MBSE,  Decision 
Making  and 

Integration 

NSWC 

Philadelphia 

new  set  of  practical  and  utilitarian  tools 
greater  efficiency  and  effectiveness 

Decision  Making 

MBSE,  Decision 
Making  and 

Integration 

WP  AFB 

developing  integrated  decision  frameworks 

Decision  Making 

MBSE,  Decision 
Making  and 

Integration 

Aerospace 

Corp 

Integrated  Decision  Making  and  Portfolio 
Management: 

Decision  Making 

MBSE,  Decision 
Making  and 

Integration 

SNL  -  1 

Digital  Engineering  -  Wish  there  was  a 
practical  notion  of  a  roadmap 

Computer  Science  Concepts 
Applications 

MBSE,  Decision 
Making  and 

Integration 

WP  AFB 

Value  of  information  theory 

Computer  Science  Concepts 
Applications 

MBSE,  Decision 
Making  and 

Integration 

IC  DoD  -  2 

Is  there  way  to  systems  engineering 
processes  to  improve  programmatic 
constraints 

Computer  Science  Concepts 
Applications 

MBSE,  Decision 
Making  and 

Integration 

IC  DoD  -  2 

challenging  the  concept  of  phases  and  gates 
-  and  with  computational  design  techniques 

Computer  Science  Concepts 
Applications 

MBSE,  Decision 
Making  and 

Integration 

JPL 

Testing  for  cyber  physical  systems 

Testing 

Security,  Trust, 

Risk  and  Testing 

AFOTEC 
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UAS  Testing 

Testing 

Security,  Trust, 

Risk  and  Testing 

AFOTEC 

System  of  Systems  Testing 

Testing 

Security,  Trust, 

Risk  and  Testing 

AFOTEC 

Testing  is  too  late  for  some  system  issues 

Testing 

Security,  Trust, 

Risk  and  Testing 

AFOTEC 

Test  optimization 

Testing 

Security,  Trust, 

Risk  and  Testing 

NSWC 

Dahl  green 

Integrating  system  security  into  systems 
engineering  practices 

Security,  Reliability  and 
Resilience 

Security,  Trust, 

Risk  and  Testing 

WP  AFB 

measure  the  security  and  resilience  of  a 
weapon  system 

Security,  Reliability  and 
Resilience 

Security,  Trust, 

Risk  and  Testing 

WP  AFB 

How  can  systems  engineering  be  made 
more  resilient 

Security,  Reliability  and 
Resilience 

Security,  Trust, 

Risk  and  Testing 

Aerospace 

Corp 

Reliability  much  more  complicated  when 
talking  about  threads 

Security,  Reliability  and 
Resilience 

Security,  Trust, 

Risk  and  Testing 

IC  DoD  -  2 

need  to  develop  an  holistic  approach  to 
cybersecurity 

Cybersecurity 

Security,  Trust, 

Risk  and  Testing 

NSWC 

Corona 

Cyber  defense  includes  a  trusted  supply 
chain. 

Cybersecurity 

Security,  Trust, 

Risk  and  Testing 

NSWC 

Corona 

Integration  of  SE  to  cyber  security 
especially  for  SoS 

Cybersecurity 

Security,  Trust, 

Risk  and  Testing 

NSWC 

Dahlgreen 

Modeling  embedded  systems  and  cyber¬ 
physical  system 

Cybersecurity 

Security,  Trust, 

Risk  and  Testing 

NAVSEA 

need  for  improvement  in  model  validation 
methods  and  approaches 

System  Trust 

Security,  Trust, 

Risk  and  Testing 

NUWC 

trusted  systems. 

System  Trust 

Security,  Trust, 

Risk  and  Testing 

JPL 

How  to  get  SoS  vulnerability  analysis  for 
prioritization  of  critical  risk 

System  Trust 

Security,  Trust, 

Risk  and  Testing 

NSWC 

Dahlgreen 

System  Security  at  Scale 

System  Trust 

Security,  Trust, 

Risk  and  Testing 

SNL-2 

System  Trust: 

System  Trust 

Security,  Trust, 

Risk  and  Testing 

SNL  -  1 

simulation  capabilities  that  they  trust  as  a 
source  of  truth  -  accepatance  need 

System  Trust 

Security,  Trust, 

Risk  and  Testing 

NUWC 

Validation,  verification  and  accreditation  of 
models  and  analysis  tools 

System  Trust 

Security,  Trust, 

Risk  and  Testing 

AMSAA 

System  definition  (reliability  modeling)  for 
complex 

System  Trust 

Security,  Trust, 

Risk  and  Testing 

AMSAA 

process  to  assess  Mean  Time  until  Out  Of 
Tolerance 

System  Trust 

Security,  Trust, 

Risk  and  Testing 

NSWC 

Corona 

error  and  fault  monitoring  are  not  able  to 
keep  up  with  the  “speed  of  operations 

System  Trust 

Security,  Trust, 

Risk  and  Testing 

IC  DoD  -  1 

Uncertainty  quantification 

Risk  and  Uncertainty 

Security,  Trust, 

Risk  and  Testing 

AMSAA 
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integrated  design  frameworks:  uncertainty 
multiple  stakeholders,  conflicting  priorities. 

Risk  and  Uncertainty 

Security,  Trust, 

Risk  and  Testing 

Aerospace 

Corp 

Risk  needs  to  be  addressed  from  a  life  cycle 
perspectiv 

Risk  and  Uncertainty 

Security,  Trust, 

Risk  and  Testing 

NSWC 

Corona 

consider  risks  in  a  more  integrated  fashion 

Risk  and  Uncertainty 

Security,  Trust, 

Risk  and  Testing 

JPL 

Movement  from  a  policy  culture  to  an 
incentive  culture 

Mission  Engineering 

Management, 
Culture  and 

Agility 

IC  DoD  -  2 

tension  building  the  optimized  architecture 
for  “my  mission”  versus  gotta  get  the  job 
done 

Mission  Engineering 

Management, 
Culture  and 

Agility 

IC  DoD  -  2 

constraints  of  the  color  of  money  and 
acquisition  lead  time 

Mission  Engineering 

Management, 
Culture  and 

Agility 

IC  DoD  -  2 

install  enterprise  architecture  and  systems 
engineering  into  established  culture 

Mission  Engineering 

Management, 
Culture  and 

Agility 

IC  DoD  -  2 

Tension  between  the  mission  thread  and 
owners  of  systems/programs. 

Mission  Engineering 

Management, 
Culture  and 

Agility 

IC  DoD  -  2 

collaborative  engineering  of  space  systems, 
be  extended  into  other  domains 

Mission  Engineering 

Management, 
Culture  and 

Agility 

Aerospace 

Corp 

spanning  multiple  languages,  doctrine,  and 
culture 

Mission  Engineering 

Management, 
Culture  and 

Agility 

Aerospace 

Corp 

How  does  agile  benefit  or  apply  to  systems 
engineering? 

Agile  Engineering 

Management, 
Culture  and 

Agility 

Aerospace 

Corp 

exploring  “quick  ugly  design  approaches”  to 
avoid  closing  down  the  design  space 

Agile  Engineering 

Management, 
Culture  and 

Agility 

NUWC 

How  to  effect  agile  engineering  re  balance 

Agile  Engineering 

Management, 
Culture  and 

Agility 

IC  DoD  -  2 

capability  to  roll  out  new  platforms  is  not  so 
good  -  platforms  requring  decades 

Development  and 

Deployment  Rapidity 

Management, 
Culture  and 

Agility 

NAWC 

Rapid  deployment  strategy  instead  ofs 
skipping  acquisition  steps 

Development  and 

Deployment  Rapidity 

Management, 
Culture  and 

Agility 

NSWC 

Corona 

accelerating  their  technology  development 
rate 

Development  and 

Deployment  Rapidity 

Management, 
Culture  and 

Agility 

NAWC 

greater  agility  in  system  development  at  the 
scale  of  the  enterprise 

Development  and 

Deployment  Rapidity 

Management, 
Culture  and 

Agility 

WP  AFB 
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Reuse  for  Agility  and  Safety: 

Development  and 

Deployment  Rapidity 

Management, 
Culture  and 

Agility 

SNL  -  1 

manage  the  tension  between  programs  of 
record  on  the  one  hand,  and  mission  threads 
on  the  other 

Project  “Lego”  Management 

Management, 
Culture  and 

Agility 

NAVSEA 

Being  able  to  work  threads  better 

Project  “Lego”  Management 

Management, 
Culture  and 

Agility 

IC  DoD  -  2 

How  to  manage  the  thread  for  operations 

Project  “Lego”  Management 

Management, 
Culture  and 

Agility 

IC  DoD  -  2 

What  are  the  resources  required  to  integrate 
the  Legos 

Project  “Lego”  Management 

Management, 
Culture  and 

Agility 

IC  DoD  -  2 

multiple  “lego  pieces”  in  diverse 
geographical  instances 

Project  “Lego”  Management 

Management, 
Culture  and 

Agility 

IC  DoD  -  1 

small  projects  over  a  short  duration;  ability 
to  invest  in  strategic  matters  is  curtailed 

Project  “Lego”  Management 

Management, 
Culture  and 

Agility 

NSWC 

Philadelphia 

cost  data  is  overly  optimistic  with  the 
“ecosystem”  accepts  it,  causing  overruns 

Other 

Other 

NSWC 

Philadelphia 

Customization 

Other 

Other 

IC  DoD  -  1 

Deployability 

Other 

Other 

IC  DoD  -  1 

All  aspects  of  unmanned  and  autonomous 
systems 

Other 

Other 

NAVSEA 

install  policies  re  admin  policies 

Other 

Other 

IC  DoD  -  2 

policy  impact:  deal  with  unintended 
consequences 

Other 

Other 

NSWC 

Philadelphia 

Systyem  Integration 

System  Integration 

MBSE,  Decision 
Making  and 

Integration 

SPAWAR 

Model  Based  Systems  Engineering 

Model  Based  SE 

MBSE,  Decision 
Making  and 

Integration 

SPAWAR 

Inherent  Uncertainties  and  Lack  of 
Continuity  in  Sensor  Data 

Risk  and  Uncertainty 

Security,  Trust, 

Risk  and  Testing 

SPAWAR 

Idiosyncracies  of  Legacy  Systems 

System  Integration 

MBSE,  Decision 
Making  and 

Integration 

SPAWAR 

Different  Paths  of  Legacy  Communications 

System  Integration 

MBSE,  Decision 
Making  and 

Integration 

SPAWAR 
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Application  of  SE  to  complex  programs 

Data/Complexity 

Knowledge,  Data 
and  Machine 

Learning 

SPAWAR 

Top  actions  to  take  to  enhance  SE  to  better 
serve  missions 

Systems  Engineering 

Modeling  and  Measure 

MBSE,  Decision 
Making  and 

Integration 

SPAWAR 

What  is  the  role  of  SE  in  analysis  of  war 
fighting  capabilities 

Systems  Engineering 

Modeling  and  Measure 

MBSE,  Decision 
Making  and 

Integration 

SPAWAR 

System  and  program  resiliency  and 
flexibility 

Security,  Reliability  and 
Resilience 

Security,  Trust, 

Risk  and  Testing 

SPAWAR 

Organizational  policy  in  Systems 

Engineering 

Mission  Engineering 

Management, 
Culture  and 

Agility 

SPAWAR 

How  to  assess  programs  and  systems 

Systems  Engineering 

Modeling  and  Measure 

MBSE,  Decision 
Making  and 

Integration 

SPAWAR 

Speed  to  capability  and  thread 

identifications 

Agile  Engineering 

Management, 
Culture  and 

Agility 

SPAWAR 

Military  Hardware  and  Equipment 

Other 

Other 

NAVSEA -  2 

Digital  Transformations 

Computer  Science  Concepts 
Applications 

MBSE,  Decision 
Making  and 

Integration 

NAVSEA -  2 

Data  Analytics 

Data  Analytics 

Knowledge,  Data 
and  Machine 

Learning 

NAVSEA -  2 

Unmanned  Systems 

Other 

Other 

NAVSEA -  2 

Artificial  intelligence 

AI/Machine  Learning 

Knowledge,  Data 
and  Machine 

Learning 

NAVSEA -  2 

Virtual  Reality 

Other 

Other 

NAVSEA -  2 

Advanced  Materials 

Other 

Other 

NAVSEA -  2 

Energy  Harvesting 

Other 

Other 

NAVSEA -  2 
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